Voxel-based, real-time acoustic adjustment

ABSTRACT

Systems, methods, and computer media for adjusting acoustics in real time are provided herein. An acoustic adjustment system can include a processor, a memory, a voxelizer, and a pathing analyzer. The voxelizer can be configured to generate a voxel-based representation of a three-dimensional (3D) interactive media environment. The voxel-based representation includes a number of individual voxels that together represent the environment. The voxelizer can also be configured to identify a plurality of blocked voxels that intersect visual elements of the 3D interactive media environment. The pathing analyzer can be configured to determine, for the clear voxels, audio path information representing a path of clear voxels between the voxel and the location of the listener. The audio path information can be used to adjust sounds emitted by sound emitters in the media environment to realistically reflect the distances the sound travels to reach the listener.

BACKGROUND

As computer software and hardware have become more powerful andadvanced, media environments, such as video game environments, havebecome richer and more realistic. Graphics, movement of characters andavatars, and the interaction of various visual elements have all becomeincreasingly realistic. Despite the advances in other aspects of mediaenvironments, however, providing realistic sound remains extremelydifficult and computationally complex. Conventional approaches tosimulating realistic sound are also typically too computationallyintense to adapt quickly to changes in the media environment.

SUMMARY

Examples described herein relate to acoustic adjustment based onvoxelized representations of 3D media environments. An acousticadjustment system can include a processor, a memory, a voxelizer, and apathing analyzer. The voxelizer can be configured to generate, by theprocessor, a voxel-based representation of a three-dimensional (3D)interactive media environment. The voxel-based representation includes aplurality of voxels, which can be of a uniform size. The voxelizer canalso be configured to store the voxel-based representation in the memoryand to identify, by the processor, a plurality of blocked voxels.Blocked voxels are voxels that intersect visual elements of the 3Dinteractive media environment. The pathing analyzer can be configured todetermine, by the processor and for the clear voxels, audio pathinformation representing a path of clear voxels between the voxel andthe location of the listener, wherein clear voxels are voxels other thanblocked voxels.

The path of clear voxels can indicate, for example, a shortest path ofsound, in the 3D interactive media environment, between a voxel and thelistener. The shortest path can account for propagation of sound aroundobstacles. In some examples, the acoustic adjustment system alsodetermines reverberation information for the location of the listenerand/or location of a sound emitter. The audio path information and/orreverberation information can be used to adjust sounds emitted by soundemitters to realistically reflect the distances the sound travels toreach the location of the listener, the characteristics of thelistener's environment, and/or the characteristics of the soundemitter's environment.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

The foregoing and other objects, features, and advantages of the claimedsubject matter will become more apparent from the following detaileddescription, which proceeds with reference to the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example acoustic adjustment system.

FIG. 2 is a diagram illustrating an example two-dimensional (2D)voxel-based representation of a media environment, the voxel-basedrepresentation indicating distance from each clear voxel to a listenerlocation.

FIG. 3 is a diagram illustrating an example 2D voxel-basedrepresentation of a media environment, the voxel-based representationindicating distance from each clear voxel to a listener location and anaudio path in which the sound travels around an obstruction to reach thelistener location.

FIG. 4 is a diagram illustrating an example 2D voxel-basedrepresentation of a media environment, the voxel-based representationindicating audio paths from sound emitters, around obstructions, to alistener location.

FIG. 5 is a diagram illustrating another example 2D voxel-basedrepresentation of a media environment, the voxel-based representationindicating audio paths from sound emitters, around obstructions, to alistener location, where the sound emitter locations and listenerlocations being different from the diagram of FIG. 4.

FIG. 6 is a diagram illustrating another example 2D voxel-basedrepresentation of a media environment, the voxel-based representationindicating audio paths from sound emitters, around obstructions, to alistener location, where two of the sound emitters are occluded.

FIG. 7 is a block diagram of an example acoustic adjustment system thatincludes an environment analyzer.

FIG. 8 is a flowchart illustrating an example acoustic adjustmentmethod.

FIG. 9 is a flowchart illustrating an example acoustic adjustment methodin which audio signals produced by sound emitters are adjusted.

FIG. 10 is a diagram of an example computing system in which somedescribed embodiments can be implemented.

FIG. 11 is an example mobile device that can be used in conjunction withthe technologies described herein.

FIG. 12 is an example cloud-supported environment that can be used inconjunction with the technologies described herein.

DETAILED DESCRIPTION

Using the systems, methods, and computer-readable media describedherein, acoustics in 3D media environments can be adjusted in real timeto accurately reflect the characteristics of the media environment.Unlike conventional attempts to realistically simulate sound in mediaenvironments, the described examples allow real-time analysis by using acomputationally efficient voxel-based representation of the 3D mediaenvironment. The described examples can also account for both thedistance sound travels, including indirect paths, as well asreverberation in the listener's immediate area and reverberation in asound emitter's immediate area, allowing a realistic acousticsimulation.

As an example, in a video game environment, spatial information (e.g.,dimensions, locations of visual elements, etc.) representing the videogame environment can be used to generate a voxel-based representation ofthe video game environment in which the volume of the game environmentis divided into a number of voxels (e.g., equally sized cubes). Thevoxel-based representation is a simplified representation of the videogame environment that allows for faster and more computationallyefficient acoustic propagation calculations. For voxels containingvisual elements (also referred to as “geometry”) of the video gameenvironment (such as walls, trees, etc.), the voxels can be designatedas “blocked” to indicate an obstruction to sound.

A clear-voxel distance, which accounts for obstruction by blockedvoxels, can be determined from the locations of sound emitters to thecurrent position of a listener. This clear-voxel distance can be used toadjust the audio generated in the video game environment to more closelycorrespond to what a listener would actually hear given the locations ofthe sound emitters and the known obstructions (blocked voxels). Forexample, if the sound takes an indirect path around obstructions, andthe indirect path is twice as far as a direct path (assuming noobstructions were present), some of the high-frequency components of thesound will be disproportionately attenuated by the increased distance ascompared to the low-frequency components of the sound.

Continuing the above example, the characteristics of the video gameenvironment can also be used to determine reverberation information fromthe listener's perspective. This can be done, for example, bydetermining a distance, from the listener, to the nearest blocked voxelin many directions, and in some examples, the type of material andcorresponding acoustic properties of the blocked voxels. The locationsof blocked voxels inform the way in which sound is reflected around thelistener. For example, if a wall in the 3D media environment(represented as blocked voxels) is six inches behind the listener, soundwill reverberate differently than if the nearest blocked voxel is 20feet behind the listener.

In the described examples, the computational complexity of determiningaudio path information (e.g., distance between sound emitters and thelistener) and reverberation is reduced through the voxel-based approach,which reduces the amount of processing power, memory, and energy neededto accurately simulate acoustics in the media environment and allows thesimulated acoustics to be updated in real time as the location of thelistener changes, the locations of sound emitters change, or thegeometry of the media environment changes. Examples are described belowwith reference to FIGS. 1-12.

FIG. 1 illustrates an acoustic adjustment system 100 implemented on oneor more computing device(s) 102. Computing device(s) 102 includesprocessor(s) 104 and a memory 106. A voxelizer 108 is configured to, byprocessor(s) 104, generate a voxel-based representation 110 of athree-dimensional (3D) interactive media environment. For example,voxelizer 108 can receive spatial information 112 representing the 3Dinteractive media environment, such as coordinate values demarcating avolume extent of the environment and coordinates or other informationindicating the location of visual elements in the environment. Voxelizer108 can then divide the volume into voxels. As used herein, voxels arevolume elements (units of volume) similar to how pixels aretwo-dimensional (2D) area elements. Voxels can be, for example, cubes ofa uniform size. Voxelizer 108 can divide a volume according to differentresolutions (i.e., different numbers of voxels per unit of measuredvolume). The 3D interactive media environment can be, for example, avideo game environment, a virtual reality environment, or an augmentedreality environment.

Voxel-based representation 110 is a simplified representation of the 3Dinteractive media environment, and different resolutions allow for moreor less accuracy (where a lower resolution results in a lowercomputational complexity). Voxelizer 108 can be configured to storevoxel-based representation 110 in memory 106. Voxelizer 108 can also beconfigured to identify, by the processor, a plurality of blocked voxels.Blocked voxels are voxels that intersect visual elements of the 3Dinteractive media environment. Information indicating the location ofvisual elements can be included in the spatial information 112 and canalso be referred to as the “geometry” or “triangles” of the environment.Voxelizer 108 can determine intersections of visual elements with voxelsby comparing the locations of the geometry with the extent of theindividual voxels. If there is an intersection, voxelizer 108 candesignate the entire voxel as blocked.

Blocked voxels can obstruct or occlude sound. As used herein,“obstruction” refers to one or more blocked voxels that prevent soundwaves from propagating along a direct path to a listener but still allowan indirect path to the listener. As used herein, “occlusion” refers toa group of blocked voxels that prevent sound from propagating to thelistener by either a direct or indirect path. An example of obstructionis a wall of a room with an open door separating a radio playing in theroom from a listener standing outside the room. The wall of the roomobstructs the most direct path for sound to travel from the radio to thelistener (through the wall), but the sound can travel through the doorand around to the listener. An example of occlusion is a radio playinginside of safe or other enclosure. In some instances, sound canpropagate through a material even though the material completelyencloses the sound. In such examples, an amount of occlusion can bedetermined by the material's audio propagation properties (also referredto as the material's audio “transparency”).

A pathing analyzer 114 can be configured to determine, by processor 104and for respective voxels of the plurality of voxels that are clearvoxels, audio path information 116 representing a path of clear voxelsbetween the voxel and a location of the listener. Clear voxels arevoxels other than blocked voxels. The path of clear voxels indicates apath of sound, in the 3D interactive media environment, between alocation corresponding to the voxel and the location of the listener.The path of clear voxels can be, for example, a shortest clear voxelpath. In some examples, pathing analyzer 114 is configured to determinemultiple clear voxel paths between individual clear voxels and thelocation of the listener, and audio path information 116 can representthe multiple paths.

Audio path information 116 can include a length of the path of clearvoxels. This is illustrated in FIGS. 2-3. FIG. 2 illustrates an example2D voxel-based representation 200 of a media environment. FIG. 2 shows a2D example for ease of illustration, which can be thought of as a volumecross section. In FIG. 2, blocked voxels are denoted with an “X.” Forclear voxels, a shortest clear-voxel distance to the listener is shown.The listener is located in voxel 202 and is denoted with an “L.” Thebolded voxels extending from voxel 204 to voxel 202 indicate a shortestclear-voxel path between voxel 204 and voxel 202.

FIG. 3 illustrates another example 2D voxel-based representation 300 ofa media environment. As in FIG. 2, In FIG. 3, blocked voxels are denotedwith an “X,” the listener is denoted with an “L” (in voxel 302), andclear voxels indicate a shortest clear-voxel distance to the listener.In FIG. 3, a number of blocked voxels create an obstruction betweenvoxel 304 and the listener at voxel 302. If the blocked voxels wereclear, a shortest path between voxels 304 and 302 would extendhorizontally between voxels 304 and 302. Because of the obstruction, theshortest clear voxel path (shown by bolded boxes) is indirect and passesthrough voxel 306. If voxel 306 were also blocked, then voxel 304 wouldbe occluded from the listener in voxel 302. The path distance shown ineach voxel is calculated using the distance between voxel centers in thepath. Accordingly, while the distance between a voxel and an adjacenthorizontal or vertical voxel is 1.0, the distance between adjacentdiagonal voxel centers, such as between voxel 308 and 310, isapproximately 1.4 (the square root of two).

Returning to FIG. 1, audio path information 116 can also include anangle from the voxel to the next voxel of the path of clear voxels. Inthis way, each voxel on the clear voxel path is associated with theangle to the next voxel in the direction of the listener. The angle canbe, for example, in the form of an x-direction, y-direction, andz-direction angle component or an x-direction, y-direction, andz-direction distance to the next voxel center.

Pathing analyzer 114 can be further configured to determine a ratio ofthe length of the path of clear voxels to a shortest distance betweenthe voxel and the location of the listener. The shortest distancecorresponds to the length of a line connecting the voxel and thelocation of the listener. The ratio indicates an audio adjustment thataccounts for the length of the path of clear voxels, and can be used todetermine an obstruction value. For example, if the clear voxel path is20, and the shortest (direct, ignoring blocked voxels) path is 10, theratio is 2:1. This can be converted to an obstruction value through avariety of approaches. In one example, obstruction values of 0.0 resultwhen the ratio is 1:1, and obstruction values of 1.0 result when theratio is infinity (or very large):1. Occlusion can be indicated withvalues of 1, for example, if a voxel is occluded, and 0 if the voxel isnot occluded.

The length of the acoustic path that audio travels can affect thefrequency composition of a received signal as compared to an emittedsignal. For example, over long distances, higher frequency components ofan audio signal tend to attenuate disproportionately to lower frequencycomponents. Given knowledge of the ratio between the clear voxel pathtraveled and what a direct path (absent blocked voxels) would have beenallows the frequencies of the audio received at the listener location tobe adjusted accordingly for realistic simulation.

Pathing analyzer 114 can determine audio path information 116 through,for example, a floodfilling approach in which after a starting voxel isexamined, an adjacent voxel is examined, followed by another, etc.,until the entire clear voxel space of voxel-based representation 110 hasbeen examined.

Pathing analyzer 114 can store audio path information 116, and anapplication such as a video game application or virtual realityapplication can provide locations of sound emitters. The voxel(s)corresponding to the sound emitter can be identified by system 100, andaudio path information for the voxel can be accessed, providing a “lookup” of audio path information given a location.

Pathing analyzer 114 can perform floodfilling or other approaches todetermine audio path information 116 for the respective clear voxelsagain periodically or upon determining that the 3D interactive mediaenvironment has changed, a sound emitter has moved, or the listener hasmoved. Frequent updates can be performed with limited computingresources because of the computational efficiency of using voxel-basedrepresentation 110.

FIGS. 4-6 illustrate additional 2D examples in which sound emitters,blocked voxels, and a listener are located. In 2D voxel-basedrepresentation 400 of FIG. 4, a listener 402 is mostly surrounded byblocked voxels (denoted by an “X”). Clear voxels are denoted with “-”,sound emitters are denoted as “E” surrounded by a box, and voxels alongthe clear-voxel path from the sound emitters to listener 402 areindicated as “c”. The grid framework shown in FIGS. 2 and 3 is omittedfor clarity, as are path distances. Because of listener 402's locationinside a rectangle of blocked voxels with only one clear voxel, voxel404, the clear voxel path for each sound emitter passes through voxel404. Voxel group 406 are occluded voxels, denoted with “o”. The occludedvoxels are entirely surrounded by blocked voxels, indicating that audiofrom the sound emitters cannot reach voxel group 406.

FIG. 5 illustrates 2D voxel-based representation 500. As in FIG. 4, “X”indicates blocked voxels, sound emitters are denoted as “E” surroundedby a box, “c” indicates voxels along the clear-voxel path from the soundemitters to listener 502, and occluded voxels are denoted by an “o”. Inrepresentation 500, sound emitted by both emitters 504 and 506 travelsthrough clear voxel 508 on the clear voxel path to listener 502.

FIG. 6 illustrates 2D voxel-based representation 600. In FIG. 6, soundemitters 602 and 604 are occluded (completely surrounded by blockedvoxels), and thus there is no clear voxel path from sound emitters 604and 602 to listener 606.

FIG. 7 illustrates a system 700 implemented on one or more computingdevice(s) 702. Computing device(s) 702 includes processor(s) 704 and amemory 706. Computing device(s) 702 can be a game console, a desktopcomputer, laptop computer, mobile device, or other computing device. Avideo game application 708 is running on computing device 702 andprovides spatial data 710 defining a 3D video game environment. Avoxelizer 712, which can be similar to voxelizer 108 of FIG. 1,generates a voxel-based representation 714 of the game environment.

A pathing analyzer 716, which can be similar to pathing analyzer 114 ofFIG. 1, determines audio path information 718, which can be similar toaudio path information 116 of FIG. 1. An environment analyzer 720 isconfigured to determine, by processor 704, a distance, from the locationof a listener, to a blocked voxel in respective directions of aplurality of directions. The distances indicate a sound reverberationadjustment to audio reaching, in the 3D interactive media environment,the location of the listener. In some examples, the nearest blockedvoxel is determined in each of 26 directions from the location of thelistener. Other numbers of directions are also contemplated. Environmentanalyzer 720 provides reverberation information 722 to video gameapplication 708 based at least in part on the distances. Reverberationinformation 722 can include the distances and can also or alternativelyinclude early reflection values and late reverberation values determinedbased on the distances.

Environment analyzer 720 can also be configured to determine, byprocessor 704, a distance, from the location of a sound emitter, to ablocked voxel in respective directions of a plurality of directions. Thedistances indicate a sound reverberation adjustment to the audioemanating, in the 3D interactive media environment, from the location ofthe sound emitter. As with the listener location, in some examples, thenearest blocked voxel is determined in each of 26 directions from thelocation of the sound emitter. Reverberation information 722 can alsoinclude the distances and/or early reflection values and latereverberation values for the sound emitter.

In some examples, environment analyzer 720 is further configured todetermine, by processor 704, a material type of the blocked voxels inthe respective directions. The material type indicates an additionalsound reverberation adjustment to audio reaching, in the 3D interactivemedia environment, the location of the listener. For example, analuminum surface reflects sound differently than a wood surface, andknowing the material type allows reverberation calculations to beadjusted. Material type information for the geometry of the game (andtherefore for the blocked voxels) can be included with the spatial data710 that defines the 3D video game environment.

In some examples system 700 also includes an audio mixer (not shown)configured to, by processor 704, adjust one or more audio signals that,in the 3D interactive environment, reach the location of the listener,the adjusting based at least in part on the audio path information. Theaudio mixer can be part of video game application 708 or can be aseparate component of system 700.

In some examples, voxelizer 712, pathing analyzer 716, and environmentanalyzer 720 are implemented as a library, plugin(s), or application(s)accessible to video game application 708 and other applications throughan application program interface (API). In such an arrangement,voxelizer 712, pathing analyzer 716, and environment analyzer 720 can beimplemented on a computing device, and different video games or virtualreality applications can access them. Pathing analyzer 716 andenvironment analyzer 720 can be configured to provide different types ofoutput (e.g., clear voxel path distance vs. ratio) depending upon theapplication interacting with voxelizer 712, pathing analyzer 716, andenvironment analyzer 720. In some examples, voxelizer 712, pathinganalyzer 716, and environment analyzer 720 are part of video gameapplication 708. In some examples, at least one of voxelizer 712,pathing analyzer 716, or environment analyzer 720 are implemented in thecloud and accessible via a cloud-based or local video game applicationor other application.

FIG. 8 illustrates a method 800 for acoustic adjustment. In processblock 802, data representing a 3D interactive media environment isreceived. In process block 804, the location of a listener within the 3Dinteractive media environment is received. In process block 806, basedon the data representing the 3D interactive media environment, avoxel-based representation of the 3D interactive media environment isgenerated. The voxel-based representation can include a plurality ofvoxels of a uniform size. In process block 808, voxels, of the pluralityof voxels, that intersect a visual element of the 3D interactive mediaenvironment are designated as blocked voxels.

For respective voxels of the plurality of voxels that are clear voxels,wherein clear voxels are voxels other than blocked voxels, audio pathinformation between the voxel and the location of the listener isdetermined in process block 810. The audio path information can includeclear voxel path distances, ratios of the clear voxel path to a directpath, angles to the next voxel along the clear voxel path, obstructionvalues, and/or occlusion values. In process block 812, soundreverberation information is determined for the location of thelistener. In process block 814, both (i) the audio path information forone or more of the respective clear voxels and (ii) the soundreverberation information are communicated to an application managingthe 3D interactive media environment (e.g., a video game or virtualreality application). Method 800 can be performed, for example, bysystem 100 of FIG. 1 and/or system 700 of FIG. 7.

FIG. 9 illustrates a method 900 for acoustic adjustment. In processblock 902, the location of a listener in a three-dimensional (3D)interactive media environment is determined. In process block 904,locations of one or more sound emitters in the 3D interactive mediaenvironment are determined. In process block 906, (i) spatialinformation for the 3D interactive media environment, (ii) the locationof the listener, and (iii) the location of the one or more soundemitters are provided to an acoustic adjustment system configured togenerate a voxel-based representation of the 3D interactive mediaenvironment and analyze the propagation of audio signals within the 3Dinteractive media environment. The acoustic adjustment system can besimilar to, for example, system 100 of FIG. 1 or system 700 of FIG. 7.In process block 908, (i) audio path information from the locations ofthe one or more sound emitters to the location of the listener and (ii)reverberation information for the location of the listener are receivedfrom the acoustic adjustment system. In process block 910, audio signalsproduced by the one or more sound emitters are adjusted based on theaudio path information and the reverberation information. Method 900 canbe performed, for example, by a computing device running a video game,virtual reality application, or augmented reality application.

In some examples, a media environment is very large and contains a largeamount of empty space. As an example, consider a first-person boatnavigating through a largely empty ocean to a distant island. In suchexamples, different portions of the game volume can be defined overwhich a voxel-based representation can be created and audio pathinformation, etc., generated, and some portions for which a voxel-basedrepresentation is not generated. To continue the above example, the“local” environment around the boat, and perhaps the environment aroundthe distant island, can be voxelized, and audio paths can be determinedin accordance with the described approaches, but much of the empty oceanenvironment is not analyzed (e.g., clear voxel paths are not determined)to save computing power. In such cases, sound reaching the edge of theisland environment can be propagated to the local boat environment usinga simple straight-line propagation path. In this example, updatesperformed as the listener moves in the media environment can trigger aredetermination of clear voxel paths for the local environment anddistant island environment rather than for the entire environment.

Example Computing Systems

FIG. 10 depicts a generalized example of a suitable computing system1000 in which the described innovations may be implemented. Thecomputing system 1000 is not intended to suggest any limitation as toscope of use or functionality, as the innovations may be implemented indiverse general-purpose or special-purpose computing systems.

With reference to FIG. 10, the computing system 1000 includes one ormore processing units 1010, 1015 and memory 1020, 1025. In FIG. 10, thisbasic configuration 1030 is included within a dashed line. Theprocessing units 1010, 1015 execute computer-executable instructions. Aprocessing unit can be a general-purpose central processing unit (CPU),processor in an application-specific integrated circuit (ASIC), or anyother type of processor. In a multi-processing system, multipleprocessing units execute computer-executable instructions to increaseprocessing power. For example, FIG. 10 shows a central processing unit1010 as well as a graphics processing unit or co-processing unit 1015.The tangible memory 1020, 1025 may be volatile memory (e.g., registers,cache, RAM), nonvolatile memory (e.g., ROM, EEPROM, flash memory, etc.),or some combination of the two, accessible by the processing unit(s).The memory 1020, 1025 stores software 1080 implementing one or moreinnovations described herein, in the form of computer-executableinstructions suitable for execution by the processing unit(s). Forexample, memory 1020, 1025 can store voxelizer 108 and/or pathinganalyzer 114 of FIG. 1 and/or voxelizer 712, pathing analyzer 716,and/or environment analyzer 720 of FIG. 7.

A computing system may have additional features. For example, thecomputing system 1000 includes storage 1040, one or more input devices1050, one or more output devices 1060, and one or more communicationconnections 1070. An interconnection mechanism (not shown) such as abus, controller, or network interconnects the components of thecomputing system 1000. Typically, operating system software (not shown)provides an operating environment for other software executing in thecomputing system 1000, and coordinates activities of the components ofthe computing system 1000.

The tangible storage 1040 may be removable or non-removable, andincludes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, orany other medium which can be used to store information and which can beaccessed within the computing system 1000. The storage 1040 storesinstructions for the software 1080 implementing one or more innovationsdescribed herein. For example, storage 1040 can store voxelizer 108and/or pathing analyzer 114 of FIG. 1 and/or voxelizer 712, pathinganalyzer 716, and/or environment analyzer 720 of FIG. 7.

The input device(s) 1050 may be a touch input device such as a keyboard,mouse, pen, or trackball, a voice input device, a scanning device, oranother device that provides input to the computing system 1000. Forvideo encoding, the input device(s) 1050 may be a camera, video card, TVtuner card, or similar device that accepts video input in analog ordigital form, or a CD-ROM or CD-RW that reads video samples into thecomputing system 1000. The output device(s) 1060 may be a display,printer, speaker, CD-writer, or another device that provides output fromthe computing system 1000.

The communication connection(s) 1070 enable communication over acommunication medium to another computing entity. The communicationmedium conveys information such as computer-executable instructions,audio or video input or output, or other data in a modulated datasignal. A modulated data signal is a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia can use an electrical, optical, RF, or other carrier.

The innovations can be described in the general context ofcomputer-executable instructions, such as those included in programmodules, being executed in a computing system on a target real orvirtual processor. Generally, program modules include routines,programs, libraries, objects, classes, components, data structures, etc.that perform particular tasks or implement particular abstract datatypes. The functionality of the program modules may be combined or splitbetween program modules as desired in various embodiments.Computer-executable instructions for program modules may be executedwithin a local or distributed computing system.

The terms “system” and “device” are used interchangeably herein. Unlessthe context clearly indicates otherwise, neither term implies anylimitation on a type of computing system or computing device. Ingeneral, a computing system or computing device can be local ordistributed, and can include any combination of special-purpose hardwareand/or general-purpose hardware with software implementing thefunctionality described herein.

For the sake of presentation, the detailed description uses terms like“determine” and “use” to describe computer operations in a computingsystem. These terms are high-level abstractions for operations performedby a computer, and should not be confused with acts performed by a humanbeing. The actual computer operations corresponding to these terms varydepending on implementation.

Example Mobile Devices

FIG. 11 is a system diagram depicting an example mobile device 1100including a variety of optional hardware and software components, showngenerally at 1102. Any components 1102 in the mobile device cancommunicate with any other component, although not all connections areshown, for ease of illustration. The mobile device can be any of avariety of computing devices (e.g., cell phone, smartphone, handheldcomputer, Personal Digital Assistant (PDA), etc.) and can allow wirelesstwo-way communications with one or more mobile communications networks1104, such as a cellular, satellite, or other network.

The illustrated mobile device 1100 can include a controller or processor1110 (e.g., signal processor, microprocessor, ASIC, or other control andprocessing logic circuitry) for performing such tasks as signal coding,data processing, input/output processing, power control, and/or otherfunctions. An operating system 1112 can control the allocation and usageof the components 1102 and support for one or more application programs1114. The application programs can include common mobile computingapplications (e.g., email applications, calendars, contact managers, webbrowsers, messaging applications), or any other computing application.The application programs 1114 can also include image recognitiontechnology implemented using convolutional neural networks.Functionality 1113 for accessing an application store can also be usedfor acquiring and updating application programs 1114.

The illustrated mobile device 1100 can include memory 1120. Memory 1120can include non-removable memory 1122 and/or removable memory 1124. Thenon-removable memory 1122 can include RAM, ROM, flash memory, a harddisk, or other well-known memory storage technologies. The removablememory 1124 can include flash memory or a Subscriber Identity Module(SIM) card, which is well known in GSM communication systems, or otherwell-known memory storage technologies, such as “smart cards.” Thememory 1120 can be used for storing data and/or code for running theoperating system 1112 and the applications 1114. Example data caninclude web pages, text, images, sound files, video data, or other datasets to be sent to and/or received from one or more network servers orother devices via one or more wired or wireless networks. The memory1120 can be used to store a subscriber identifier, such as anInternational Mobile Subscriber Identity (IMSI), and an equipmentidentifier, such as an International Mobile Equipment Identifier (IMEI).Such identifiers can be transmitted to a network server to identifyusers and equipment.

The mobile device 1100 can support one or more input devices 1130, suchas a touchscreen 1132, microphone 1134, camera 1136, physical keyboard1138 and/or trackball 1140 and one or more output devices 1150, such asa speaker 1152 and a display 1154. Other possible output devices (notshown) can include piezoelectric or other haptic output devices. Somedevices can serve more than one input/output function. For example,touchscreen 1132 and display 1154 can be combined in a singleinput/output device.

The input devices 1130 can include a Natural User Interface (NUI). AnNUI is any interface technology that enables a user to interact with adevice in a “natural” manner, free from artificial constraints imposedby input devices such as mice, keyboards, remote controls, and the like.Examples of NUI methods include those relying on speech recognition,touch and stylus recognition, gesture recognition both on screen andadjacent to the screen, air gestures, head and eye tracking, voice andspeech, vision, touch, gestures, and machine intelligence. Otherexamples of a NUI include motion gesture detection usingaccelerometers/gyroscopes, facial recognition, 3D displays, head, eye,and gaze tracking, immersive augmented reality and virtual realitysystems, all of which provide a more natural interface, as well astechnologies for sensing brain activity using electric field sensingelectrodes (EEG and related methods). Thus, in one specific example, theoperating system 1112 or applications 1114 can comprisespeech-recognition software as part of a voice user interface thatallows a user to operate the device 1100 via voice commands. Further,the device 1100 can comprise input devices and software that allows foruser interaction via a user's spatial gestures, such as detecting andinterpreting gestures to provide input to a gaming application.

A wireless modem 1160 can be coupled to an antenna (not shown) and cansupport two-way communications between the processor 1110 and externaldevices, as is well understood in the art. The modem 1160 is showngenerically and can include a cellular modem for communicating with themobile communication network 1104 and/or other radio-based modems (e.g.,Bluetooth 1164 or Wi-Fi 1162). The wireless modem 1160 is typicallyconfigured for communication with one or more cellular networks, such asa GSM network for data and voice communications within a single cellularnetwork, between cellular networks, or between the mobile device and apublic switched telephone network (PSTN).

The mobile device can further include at least one input/output port1180, a power supply 1182, a satellite navigation system receiver 1184,such as a Global Positioning System (GPS) receiver, an accelerometer1186, and/or a physical connector 1190, which can be a USB port, IEEE1394 (FireWire) port, and/or RS-232 port. The illustrated components1102 are not required or all-inclusive, as any components can be deletedand other components can be added.

Example Cloud-Supported Environments

FIG. 12 illustrates a generalized example of a suitable cloud-supportedenvironment 1200 in which described embodiments, techniques, andtechnologies may be implemented. In the example environment 1200,various types of services (e.g., computing services) are provided by acloud 1210. For example, the cloud 1210 can comprise a collection ofcomputing devices, which may be located centrally or distributed, thatprovide cloud-based services to various types of users and devicesconnected via a network such as the Internet. The implementationenvironment 1200 can be used in different ways to accomplish computingtasks. For example, some tasks (e.g., processing user input andpresenting a user interface) can be performed on local computing devices(e.g., connected devices 1230, 1240, 1250) while other tasks (e.g.,storage of data to be used in subsequent processing) can be performed inthe cloud 1210.

In example environment 1200, the cloud 1210 provides services forconnected devices 1230, 1240, 1250 with a variety of screencapabilities. Connected device 1230 represents a device with a computerscreen 1235 (e.g., a mid-size screen). For example, connected device1230 can be a personal computer such as desktop computer, laptop,notebook, netbook, or the like. Connected device 1240 represents adevice with a mobile device screen 1245 (e.g., a small size screen). Forexample, connected device 1240 can be a mobile phone, smart phone,personal digital assistant, tablet computer, and the like. Connecteddevice 1250 represents a device with a large screen 1255. For example,connected device 1250 can be a television screen (e.g., a smarttelevision) or another device connected to a television (e.g., a set-topbox or gaming console) or the like. One or more of the connected devices1230, 1240, 1250 can include touchscreen capabilities. Touchscreens canaccept input in different ways. For example, capacitive touchscreensdetect touch input when an object (e.g., a fingertip or stylus) distortsor interrupts an electrical current running across the surface. Asanother example, touchscreens can use optical sensors to detect touchinput when beams from the optical sensors are interrupted. Physicalcontact with the surface of the screen is not necessary for input to bedetected by some touchscreens. Devices without screen capabilities alsocan be used in example environment 1200. For example, the cloud 1210 canprovide services for one or more computers (e.g., server computers)without displays.

Services can be provided by the cloud 1210 through service providers1220, or through other providers of online services (not depicted). Forexample, cloud services can be customized to the screen size, displaycapability, and/or touchscreen capability of a particular connecteddevice (e.g., connected devices 1230, 1240, 1250).

In example environment 1200, the cloud 1210 provides the technologiesand solutions described herein to the various connected devices 1230,1240, 1250 using, at least in part, the service providers 1220. Forexample, the service providers 1220 can provide a centralized solutionfor various cloud-based services. The service providers 1220 can manageservice subscriptions for users and/or devices (e.g., for the connecteddevices 1230, 1240, 1250 and/or their respective users). The cloud 1210can store voxel-based representations 1260 used in acoustic adjustmentas described herein.

Example Implementations

Although the operations of some of the disclosed methods are describedin a particular, sequential order for convenient presentation, it shouldbe understood that this manner of description encompasses rearrangement,unless a particular ordering is required by specific language set forthbelow. For example, operations described sequentially may in some casesbe rearranged or performed concurrently. Moreover, for the sake ofsimplicity, the attached figures may not show the various ways in whichthe disclosed methods can be used in conjunction with other methods.

Any of the disclosed methods can be implemented as computer-executableinstructions or a computer program product stored on one or morecomputer-readable storage media and executed on a computing device(e.g., any available computing device, including smart phones or othermobile devices that include computing hardware). Computer-readablestorage media are any available tangible media that can be accessedwithin a computing environment (e.g., one or more optical media discssuch as DVD or CD, volatile memory components (such as DRAM or SRAM), ornonvolatile memory components (such as flash memory or hard drives)). Byway of example and with reference to FIG. 10, computer-readable storagemedia include memory 1020 and 1025, and storage 1040. By way of exampleand with reference to FIG. 11, computer-readable storage media includememory and storage 1120, 1122, and 1124. The term computer-readablestorage media does not include signals and carrier waves. In addition,the term computer-readable storage media does not include communicationconnections (e.g., 1070, 1160, 1162, and 1164).

Any of the computer-executable instructions for implementing thedisclosed techniques as well as any data created and used duringimplementation of the disclosed embodiments can be stored on one or morecomputer-readable storage media. The computer-executable instructionscan be part of, for example, a dedicated software application or asoftware application that is accessed or downloaded via a web browser orother software application (such as a remote computing application).Such software can be executed, for example, on a single local computer(e.g., any suitable commercially available computer) or in a networkenvironment (e.g., via the Internet, a wide-area network, a local-areanetwork, a client-server network (such as a cloud computing network), orother such network) using one or more network computers.

For clarity, only certain selected aspects of the software-basedimplementations are described. Other details that are well known in theart are omitted. For example, it should be understood that the disclosedtechnology is not limited to any specific computer language or program.For instance, the disclosed technology can be implemented by softwarewritten in C++, Java, Perl, JavaScript, Adobe Flash, or any othersuitable programming language. Likewise, the disclosed technology is notlimited to any particular computer or type of hardware. Certain detailsof suitable computers and hardware are well known and need not be setforth in detail in this disclosure.

Furthermore, any of the software-based embodiments (comprising, forexample, computer-executable instructions for causing a computer toperform any of the disclosed methods) can be uploaded, downloaded, orremotely accessed through a suitable communication means. Such suitablecommunication means include, for example, the Internet, the World WideWeb, an intranet, software applications, cable (including fiber opticcable), magnetic communications, electromagnetic communications(including RF, microwave, and infrared communications), electroniccommunications, or other such communication means.

The disclosed methods, apparatus, and systems should not be construed aslimiting in any way. Instead, the present disclosure is directed towardall novel and nonobvious features and aspects of the various disclosedembodiments, alone and in various combinations and sub combinations withone another. The disclosed methods, apparatus, and systems are notlimited to any specific aspect or feature or combination thereof, nor dothe disclosed embodiments require that any one or more specificadvantages be present or problems be solved.

The technologies from any example can be combined with the technologiesdescribed in any one or more of the other examples. In view of the manypossible embodiments to which the principles of the disclosed technologymay be applied, it should be recognized that the illustrated embodimentsare examples of the disclosed technology and should not be taken as alimitation on the scope of the disclosed technology.

We claim:
 1. An acoustic adjustment system, comprising: a processor; amemory; a voxelizer configured to: generate, by the processor, avoxel-based representation of a three-dimensional (3D) interactive mediaenvironment, the voxel-based representation comprising a plurality ofvoxels, store the voxel-based representation of the 3D interactive mediaenvironment in the memory, and identify, by the processor, a pluralityof blocked voxels, wherein blocked voxels are voxels that intersectvisual elements of the 3D interactive media environment; and a pathinganalyzer configured to determine, by the processor and for respectivevoxels of the plurality of voxels that are clear voxels, audio pathinformation representing a path of clear voxels between the voxel and alocation of the listener, wherein clear voxels are voxels other thanblocked voxels.
 2. The system of claim 1, wherein the path of clearvoxels indicates a path of sound, in the 3D interactive mediaenvironment, between a location corresponding to the voxel and thelocation of the listener.
 3. The system of claim 1, wherein the 3Dinteractive media environment is at least one of a video gameenvironment, a virtual reality environment, or an augmented realityenvironment.
 4. The system of claim 1, wherein the system is part of avideo game console, and wherein the pathing analyzer is configured toprovide audio path information for a voxel in response to a request froma video game application.
 5. The system of claim 1, wherein the path ofclear voxels is a shortest clear-voxel path between the location of thelistener and the voxel.
 6. The system of claim 1, wherein the audio pathinformation comprises an angle from the voxel to the next voxel of thepath of clear voxels.
 7. The system of claim 1, wherein the audio pathinformation comprises a length of the path of clear voxels.
 8. Thesystem of claim 7, wherein the pathing analyzer is further configured todetermine a ratio of the length of the path of clear voxels to ashortest distance between the voxel and the location of the listener,wherein the shortest distance corresponds to the length of a lineconnecting the voxel and the location of the listener, and wherein theratio indicates an audio adjustment that accounts for the length of thepath of clear voxels.
 9. The system of claim 1, wherein the pathinganalyzer is further configured to determine, by the processor and forthe respective voxels of the plurality of voxels that are clear voxels,audio path information representing one or more additional paths ofclear voxels between the voxel and the location of the listener.
 10. Thesystem of claim 1, further comprising an environment analyzer configuredto determine, by the processor, a distance, from at least one of thelocation of the listener or a location of a sound emitter, to a blockedvoxel in respective directions of a plurality of directions, wherein thedistances indicate a sound reverberation adjustment to audio reaching,in the 3D interactive media environment, the location of the listener,or to audio originating, in the 3D interactive media environment, fromthe sound emitter.
 11. The system of claim 10, wherein the environmentanalyzer is further configured to determine, by the processor, amaterial type of the blocked voxels in the respective directions,wherein the material type indicates an additional sound reverberationadjustment to audio reaching, in the 3D interactive media environment,the location of the listener.
 12. The system of claim 1, furthercomprising an audio mixer configured to, by the processor, adjust one ormore audio signals that, in the 3D interactive environment, reach thelocation of the listener, the adjusting based at least in part on theaudio path information.
 13. One or more computer-readable storage mediastoring computer-executable instructions for acoustic adjustment, theadjustment comprising: receiving data representing a three-dimensional(3D) interactive media environment; receiving a location of a listenerwithin the 3D interactive media environment; based on the datarepresenting the 3D interactive media environment, generating avoxel-based representation of the 3D interactive media environment, thevoxel-based representation comprising a plurality of voxels of a uniformsize; designating voxels, of the plurality of voxels, that intersect avisual element of the 3D interactive media environment as blockedvoxels; for respective voxels of the plurality of voxels that are clearvoxels, wherein clear voxels are voxels other than blocked voxels,determining audio path information between the voxel and the location ofthe listener; determining sound reverberation information for thelocation of the listener; and communicating (i) the audio pathinformation for one or more of the respective clear voxels and (ii) thesound reverberation information to an application managing the 3Dinteractive media environment.
 14. The one or more computer-readablestorage media of claim 13, wherein the 3D interactive media environmentis a video game environment, and wherein the application is a videogame.
 15. The one or more computer-readable storage media of claim 13,wherein the audio path information comprises at least one of anobstruction value or an occlusion value.
 16. The one or morecomputer-readable storage media of claim 13, wherein the one or moreclear voxels for which the audio path information and soundreverberation information are communicated are locations of soundemitters in the 3D interactive media environment.
 17. The one or morecomputer-readable storage media of claim 13, wherein the determiningsound reverberation information comprises determining a distance to ablocked voxel in respective directions of a plurality of directions,wherein the distances indicate a sound reverberation adjustment to audioreaching, in the 3D interactive media environment, the location of thelistener.
 18. The one or more computer-readable storage media of claim13, wherein audio path information and sound reverberation informationare determined for clear voxels within one or more areas and notdetermined for clear voxels outside of the one or more areas.
 19. Amethod for acoustic adjustment, the method comprising: determining alocation of a listener in a three-dimensional (3D) interactive mediaenvironment; determining locations of one or more sound emitters in the3D interactive media environment; providing (i) spatial information forthe 3D interactive media environment, (ii) the location of the listener,and (iii) the location of the one or more sound emitters to an acousticadjustment system configured to generate a voxel-based representation ofthe 3D interactive media environment and analyze the propagation ofaudio signals within the 3D interactive media environment; receiving,from the acoustic adjustment system, (i) audio path information from thelocations of the one or more sound emitters to the location of thelistener and (ii) reverberation information for the location of thelistener; and adjusting audio signals produced by the one or more soundemitters based on the audio path information and the reverberationinformation.
 20. The method of claim 19, wherein the audio pathinformation reflects, for respective voxels, of the voxel-basedrepresentation, in which the one or more sound emitters are located, adistance of a path of clear voxels between the voxel and the location ofthe listener, and wherein the reverberation information comprises adistance from the location of the listener, to a blocked voxel inrespective directions of a plurality of directions.